1. Field of the Art
The present disclosure pertains to a means of providing timestamps for frames of media stream input data that are captured by a computer system.
2. Related Art
A timestamp can be a recording of the measurement of the current time of an event, as used in a computer system. There can be one or more timelines, or time bases, along which the current time of an event can be measured. There can also be many choices for how a measured timestamp is expressed. A timestamp can be expressed as a measurement along the time base it first measured; it can also be expressed as a measurement along a time base that differs from the first.
One field of use for timestamps in combination with associated captured media stream data is synchronization for playback of a media stream or multiple media streams. A computer system can be used to develop and/or implement a relationship between a time base or timeline for playback, and the timestamps recorded and associated with captured media stream data. This relationship between recorded timestamps and the playback timebase can be used advantageously to position and “play back” the captured data with synchronization.
Playback synchronization can be useful for playback of more than one captured media stream concurrently. As an example, consider the situation where a computer system is used to concurrently play back a captured audio (sound) media stream and a captured video (image) media stream that have a preferred synchronization relationship. This preferred relationship can be the “lip synch” of commercial motion pictures, that is, movies. The experience of the playback audience can be degraded both by certain ranges of fixed time offsets between the two streams, and, by variability or “slip” in synchronization during the audience playback experience. There is a need to maintain a particular fixed timing relationship between the playback media streams, within a limited tolerance and with a limited variability.
Providing timestamps with captured media stream data can also be useful for the purposes of editing and/or analysis, for one or more captured media streams.
Reference herein to captured and/or recorded values, data, media streams, and media stream data can refer to information reduced to tangible or intangible means including volatile or temporary representations within computer systems.
A lack of accuracy in a timestamp measurement during media stream data capture can limit the performance of later processes that take the timestamp into account. Thus, high accuracy is a desirable feature of timestamps.
In some computer systems, a timestamp for a frame of captured media stream input data can be determined as follows. The processor receives an indication of a frame capture event through interrupt, polling, signal, or other means, and responds by taking a direct measurement of the current time with respect to the processor. This measurement can be as simple as the processor reading a register representing the current time from a real-time-clock subsystem or another subsystem; those subsystems not being directly responsive to the frame capture event. Problems associated with this method can include a total delay comprising contributory delays that are known and unknown, contributory delays that are fixed and variable, and contributory delay between the frame capture event and the processor response, hence a delay in taking the measurement. Some of the sources of contributory delays can be a result of the operating characteristics of the operating system, and, some of the sources of contributory delays may be a result of hardware and general system characteristics. The contributory delays and total delay can limit the accuracy of the timestamp measurement.
In some computer systems, a means of providing an accurate timestamp for a frame of captured media stream input data is provided as follows. A subsystem responds directly to a frame capture event and substantially immediately records or registers a timestamp measurement. The timestamp measurement can occur without need for the processor to respond between the frame capture event and the recording. For example, consider a computer system comprising a digital video capture subsystem that can capture media stream input that is digital video. The digital video can be formatted and interconnected as described in International Telecommunications Union standards ITU-R BT.601 and/or ITU-R BT.656. In these standards there can be a specific position within the media stream, and hence a specific time within a frame of digital video, corresponding to the first onset of vertical blanking within a frame of digital video. This subsystem can operate in combination with a digital audio capture subsystem that can capture media stream input that is digital audio representing sampled analog audio. The clock rate for captured digital audio samples can be 44.1 kHz or 48 kHz or another convenient rate. The audio capture subsystem can maintain an audio sample counter, corresponding to an audio timebase, that increments regularly and precisely with each captured audio sample. The digital video capture subsystem can detect the first onset of vertical blanking event that occurs during a frame of the captured media stream of digital video. The subsystems in combination can then register and/or record the current value of the audio sample counter at the time of the vertical blanking event. The registered and/or recorded value of the audio sample counter can be a timestamp, expressed along and in terms of an audio sampling timebase, for the frame of captured digital video. Within the computer system, this timestamp can be associated with the frame of captured digital video. The accuracy of this timestamp can be sufficient to synchronize a video frame to a particular audio sample time. That is, the video frame timestamp can be audio sample-accurate.
Some software architectures operating in some computer systems can require that a timestamp that is stored and/or associated with a frame of captured media stream data be expressed in terms of a specific timebase. This specific timebase can be associated with an arbitrary clock object. The specific timebase of this requirement can be a distinctly different timebase from the timebase used to first measure and register and/or record a timestamp for a frame of captured media stream input data.
For example, consider the previously described means of providing an accurate timestamp. In some software architectures operating in some computer systems there can be a requirement that the audio sample-accurate timestamp for a video frame be expressed in terms of system time. System time can be, for example, a timebase that is of a different frequency and/or phase than any of the timebases upon which the timestamp for a captured media stream data frame is initially measured. Each timebase of interest within the computer system can have an associated clock counter whose value regularly increments responsive to the period, fraction of a period, or multiple of a period, of a distinct clock signal and/or frequency associated with each timebase. Each of these clock counters can be started, stopped, and/or set to a particular value, independently, and at different moments. The relationship between each of these clock counters can be deterministic, but in general the relationship between them is not explicitly known.
Thus, there is a difficulty in the combination of obtaining an accurate timestamp value in terms of one timebase, and a system requirement to express the timestamp value in terms of another timebase.
Accordingly, in a computer system having a media stream input, an alternate means of providing an accurate timestamp for captured frames of media stream input data is needed.